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Abstract 

Geographic  information  systems  (GIS)  timely  and  precise  spatial  information  for 
command  and  control  systems.  GIS  support  decision  makers  in  formulating  plans  and 
making  critical  decisions  in  preparedness,  response,  recovery,  and  mitigation  phase  of 
crisis  management.  The  study  developed  a  validated  and  prioritized  reference  decision¬ 
making  model  by  Panel  of  experts  from  Federal  Emergency  Management  Agency 
(FEMA)  that  measures  GIS  effectiveness  in  crisis  management.  The  model  consists  of 
seven  factors  and  forty  sub-factors. The  seven  GIS  factors  with  respect  to  their  importance 
in  ascending  order  were  system  quality,  information  quality,  user  satisfaction,  system  use, 
decision  performance,  task  complexity  and  feedback,  and  organizational  impact. 

The  data  on  measuring  GIS  effectiveness  were  obtained  from  a  survey.  Analyzed 
results  from  GIS  directors,  project  managers,  technical  staff  and  users  indicated  that  user 
satisfaction  was  assessed  the  lowest  factor  in  the  model.  Additionally,  the  lowest 
assessments  of  the  model  sub-factors  were;  error  recovery,  documentation  of  system  and 
procedures,  ease  of  learning,  currency  of  output,  top  management  involvement,  training 
provided  to  user,  and  GIS  organizational  position.  The  highest  assessments  in  the  model 
sub-factor  were  given  to  presentation  mapping,  viewing  the  map,  and  productivity 
improved  by  GIS. 

The  major  recommendations  made  include  the  following:  (1)  regularly  measure 
the  GIS  effectiveness  for  each  crisis  events  and  establish  a  backlog;  (2)  consider  some 
other  measurements  of  technical,  individual,  and  organizational  factors  based  on  GIS 
future  capabilities,  crisis  context,  and  organization  needs. 

Keywords:  Geographic  information  systems  (GIS),  crisis  management,  system  quality, 
information  quality,  user  satisfaction,  system  use,  decision  performance,  task  complexity 
and  feedback,  and  organizational  impact. 
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1.  Introduction 

We  are  able  to  witness  an  age  of  internationally  and  nationally  growing  crises. 
The  increasing  number  of  crises  is  happening  because  of  the  population  increase,  more 
people  using  urban  high-risk  areas,  geological  effects,  meteorological  changes,  expanding 
infrastructure  complexity,  increasing  threat  of  terrorism,  and  the  complexity  and  value  of 
advanced  technology  and  its  integration.  Natural  and  man-made  disasters  can  all  be  forms 
of  crisis. 

There  are  different  fonns  of  natural  disasters  such  as  tornadoes,  earthquakes, 
floods,  volcanoes  and  forest  fires.  Environmental  problems  and  landslides  could  also  be 
considered  as  natural  disasters.  Usually  natural  disasters  cause  great  loss  in  terms  of  cost 
and  people.  Human-made  disasters  could  be  either  accidental  (technological)  which  may 
include  construction  failures,  space,  aviation  and  biological  disasters,  along  with 
industrial  chemical  accidents,  and  infonnation  technology,  or  intentional  incidents  which 
may  include  terrorist  actions  or  military  conflict.  A  crisis  may  happen  at  any  moment 
with  or  without  any  warning.  A  crisis  may  shake  the  political  stability  of  a  government 
or  may  destroy  a  private  organization. 

While  disasters  or  crises  may  not  be  avoidable,  they  can  be  predicted,  or  their 
impact  can  be  minimized.  Timely  and  accurate  spatial  information  produced  by 
geographic  information  systems  (GIS)  are  the  keys  to  effective  crisis  management  [1]. 
United  States  Geological  Survey  (USGS)  [2]  defined  GIS  as  “system  of  computer 
hardware,  software  and  procedures  designed  to  support  the  capture,  management, 
manipulation,  analysis,  modeling  and  display  of  spatially  referenced  data  for  solving 
complex  planning  and  management  problems.” 

Using  GIS  in  command  and  control  systems  can  enhance  this  information  to 
reduce  loss  of  lives,  minimize  the  amount  of  crisis  damage  and  save  crisis  recovery  costs. 
GIS  tools  apply  to  the  four  phases  of  crisis  management.  These  phases  are  preparedness, 
response,  recovery  and  mitigation.  Table  1  illustrates  The  GIS  applications  for  each 
phase.  Moreover,  GIS  can  monitor  and  track  the  path  of  a  disaster,  and  forecast  potential 
problems  with  utilities  or  residential  areas  that  could  be  threatened  in  upcoming  days. 

Table  1.  GIS  Applications  &  Time  Requirements  in  Crisis  Management  Phases 


Preparedness 

Response 

Recovery 

Mitigation 

TO  TO  +72  hours 

-  Preposition 
resources 

-  Evacuation  routs 

-  Damage 
assessment 

-  Integration  with 
contingency  plan 

-  Critical  zones  & 
estimates 

-  Emergency  routs 

-  Alternative  routs 

-  Restricted  zones 
classification 

-  Demographic 
distribution 

-  Resources  sites 
updates 

-  Projects 
monitoring 

-  Risk  assessment 

-  Track  &  analyze 
incident 

-  Damage  incidents 
areas 

-  Clean  up  and 
rebuild 

The  critical  problem  in  GIS  practice  is  a  failure  to  develop  a  valid  model  to 
measure  GIS  effectiveness  for  crisis  applications.  Therefore,  based  on  a  comprehensive 
relevant  review  of  literature,  the  problem  addressed  in  this  study  is: 
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There  is  a  need  to  investigate  multidimensional  measurements  of  the  geographic 
information  systems  (GIS)  effectiveness  in  all  phases  of  crisis  management  that 
incorporates  technical,  as  well  as  individual  and  organizational  factors.  Moreover,  there  is 
a  need  to  develop  a  more  validated  decision  making  model  to  prioritize  the 
multidimensional  GIS  effectiveness  factors,  and  to  customize  the  model  for  measuring 
GIS  effectiveness  in  crisis  management. 

In  order  to  investigate  the  main  research  problem,  the  following  research 
procedure  are  identified  and  shown  in  figure  1 : 

1 .  Develop  a  theoretical  based  model  of  GIS  effectiveness  in  crisis  management  that 
combines  technical,  as  well  as  individual  and  organizational  factors. 

2.  Structure  the  developed  model  in  a  decision  making  model  for  validating  and 
prioritizing  the  GIS  effectiveness  factors. 

3.  Measure  and  assess  the  GIS  effectiveness  factors  in  crisis  management  field  using 
the  model. 

4.  Investigate  any  significant  differences  in  the  GIS  effectiveness  in  crisis 
management,  using  the  model  factors  as  dependant  variables,  through  the 
following  formulated  research  hypotheses. 

Research  Hypothesis  1:  There  are  significant  differences  in  GIS  effectiveness  in 
crisis  management  among  GIS  directors  and  project  managers,  GIS  technical 
staff,  and  GIS  end  users. 

Research  Hypothesis  2:  There  are  significant  differences  in  GIS  effectiveness  in 
crisis  management  among  GIS  users  based  on  their  years  of  work  experience. 
Research  Hypothesis  3:  There  are  significant  differences  in  GIS  effectiveness  in 
crisis  management  among  GIS  users  based  on  duration  of  training. 

Figure  1.  Research  Procedure 


Stage  II 

Stage  III 

Stage  IV 
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2.  GIS  Effectiveness  Model 

There  is  a  very  limited  research  on  the  evaluation  of  geographic  information 
systems  effectiveness.  Since  GIS  are  spatial  information  systems,  a  comprehensive 
survey  of  literature  on  the  evaluation  of  infonnation  systems  effectiveness  will  add  great 
value  in  developing  a  model  for  measuring  GIS  effectiveness. 

The  literature  review  is  the  primary  source  for  identifying  evaluation  criteria 
associated  with  the  effectiveness  of  GIS  application  in  crisis  management.  Though  the 
focus  of  this  study  is  on  GIS  effectiveness  in  crisis  management,  the  evaluation  criteria 
for  measuring  the  effectiveness  of  information  systems  will  be  used  accompanied  with 
specific  criteria  for  GIS  [3]  [4]  ]  [5]  ]  [6].  Moreover,  an  adopted  definition  for  measuring 
GIS  effectiveness  is  necessary  for  this  study.  The  USGS  defined  GIS  effectiveness  based 
on  the  definition  of  the  General  Accounting  Office  (GAO)  [2]  for  information  system 
effectiveness  which  is  “system  effectiveness  is  measured  by  determining  whether  the 
system  perfonns  the  intended  function  and  whether  users  get  the  information  they  need, 
in  the  right  form,  in  a  timely  fashion.” 

In  1949,  Shannon  and  Weaver  defined  communication  theory  levels,  which 
demonstrate  that  information  in  communication  systems  can  be  measured  at  different 
levels.  Those  levels  include  the  technical  level,  the  semantic  level  and  the  effectiveness 
level.  The  technical  level  is  the  accuracy  and  efficiency  of  the  system  which  produces  the 
information,  the  semantic  mode  is  conveying  the  information  to  meaning,  and  the 
effectiveness  level  is  the  effect  of  the  information  on  the  receiver  [7]. 

Later  in  1978,  Mason  had  adopted  the  communication  theory  to  the  infonnation 
system  field.  He  defined  effectiveness  as  influence  that  consists  of  three  levels.  Those 
three  levels  are  the  receipts  of  the  information,  the  influence  on  recipient  by  evaluating 
the  infonnation  and  its  application,  and  the  influence  on  the  system  perfonnance  through 
a  change  in  recipient  behavior  [6]  [7]. 

In  1992,  Delone  and  McLean  [7]  reviewed  a  total  of  180  research  articles  that 
proposed  different  aspects  of  information  systems  success  or  effectiveness  with  different 
measurements.  Their  contribution  to  the  information  systems  field  was  the  introduction  of 
a  taxonomy  that  presents  researchers  with  an  integrated  view  of  the  dependent  variable 
for  infonnation  systems  success.  They  proposed  a  model  of  six  major  dimensions  for 
measuring  information  systems  success.  The  six  measurements  are  system  quality, 
information  quality,  use,  user  satisfaction,  individual  impact  and  organizational  impact. 
The  dimensions  are  interdependent  on  each  other  and  researchers  are  advised  to  combine 
measures  from  each  dimension  to  form  comprehensive  instruments  for  measuring 
information  systems  success.  Moreover,  they  noted  that  “once  this  expanded  view  of 
information  systems  success  is  recognized,  it  is  not  surprising  to  find  that  there  are  so 
many  different  measures  of  this  success  in  the  literature,  depending  upon  which  aspect  of 
information  systems  the  researcher  has  focused  his  or  her  attention.  Some  of  these 
measures  have  been  merely  identified,  but  never  used  empirically.  Others  have  been 
used,  but  have  employed  different  measurements,  making  comparisons  among  studies 
difficult”  [7]  . 

There  are  many  reasons  to  indicate  that  Delone  and  McLean’s  model  is  a  strong 
and  well-developed  model  for  measuring  infonnation  systems  effectiveness.  First,  it  is 
built  based  on  a  standard  theoretical  basis.  It  was  based  on  Shannon  and  Weaver's  1949 
communication  theory  and  the  modification  done  by  Mason,  1978,  as  discussed  above. 
Second,  it  provides  a  multidimensional  approach  for  independent  variables  of 
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Information  Systems.  This  is  based  on  their  thorough  research  for  180  articles  of 
information  systems  success.  Third,  some  researchers  have  applied  this  model,  or  part  of 
the  model.  They  indicated  some  success  and  validity  of  their  results  [8]  [9]. 

Though  a  number  of  effectiveness  measures  have  been  identified  in  the  literature 
under  various  names,  some  studies  do  not  fit  exactly  into  any  one  dimension  of  the 
Delone  and  Mclean  model.  Arnold  [10]  has  modified  Delone  and  McLean’s  model.  This 
modification  was  based  on  some  criticism  of  some  research  studies  that  apply  to  Delone 
and  McLean’s  model  or  part  of  that  model.  The  modified  model  is  shown  on  Figure  2. 


Figure  2.  Geographic  Infonnation  Systems  (GIS)  Effectiveness  Model 
Source:  Vicky  Arnold,  “Discussion  of  an  Experimental  Evaluation  of 
Measurements  of  Information  System  Effectiveness,”  Journal  of  Information 
Systems,  vol  9,  no.  2  (fall  1995):  85-91. 


Arnold  added  one  factor  which  consists  of  task  complexity  and  feedback.  This  is 
an  important  extension  to  Delone  and  McLean’s  model  and  adds  significant  contribution 
to  infonnation  systems  literature  [10].  Moreover,  Arnold’s  model  shows  that  the 
individual  impact  factor  is  closely  related  to  and  is  represented  by  decision  perfonnance 
which  is  the  degree  to  which  the  system  supports  or  improves  decision  making.  In  other 
words,  it  is  the  effect  of  the  system  on  the  perfonnance  outcome  for  an  individual 
decision-maker,  or  its  collective  effects  on  the  department  or  the  organization. 

The  unique  attempt  of  this  research  is  to  adopt  Arnold’s  model  to  measure  (GIS) 
effectiveness.  The  seven  integrated  factors  of  this  model  that  will  be  used  are: 
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•  System  Quality:  is  the  measurement  of  the  infonnation  processing  system.  It  is  used 
to  assess  the  performance  of  the  information  systems. 

•  Information  Quality:  focuses  on  the  quality  of  information  systems  produced  output. 

•  System  Usage:  is  the  extent  and  nature  of  the  use  of  the  output  of  an  information 
system. 

•  User  Satisfaction:  is  the  degree  to  which  the  system  users  are  satisfied  with 
information  system. 

•  Individual  Decision  Making:  is  the  effect  of  infonnation  on  the  behavior  of  the 
recipient. 

•  Task  Complexity  and  Feedback:  is  the  degree  to  which  work  to  be  performed  is 
difficult  to  understand  and  uncertain  [11].  Feedback  is  the  loop  process  from  past 
behavior  to  current  attitudes  and  beliefs  [12]. 

•  Organizational  Impact:  represents  the  effect  of  information  that  is  produced  by  an 
information  system  on  organizational  performance. 

After  identifying  the  seven  factors  that  will  measure  GIS  effectiveness,  the  next 
step  is  to  identify  the  sub-factors  or  attributes  for  each  factor.  Those  sub-factors  are 
collected  from  the  previous  research  studies  for  information  systems  evaluation  and  from 
GIS  feature  [3]  [13]  [14]  [15]  [16]  [17].  The  organizing  of  those  sub-factors  and  their 
definitions  are  explained  below. 

3.  GIS  Effectiveness  Model  Validation  and  Prioritization  Exercise 

In  this  second  stage  of  research  design,  the  theoretical  model  was  structured  in  a 
decision  making  model  for  validation  and  prioritization  exercise  with  Federal  Emergency 
Management  Agency  (FEMA)  panel  of  experts  using  group  decision  support  software 
called  Team  Expert  Choice.  The  researcher  selected  Team  Expert  Choice  software  to 
develop  a  multi-criteria  decision-making  model  of  GIS  effectiveness  in  crisis 
management.  Team  Expert  Choice  model  employs  the  Analytical  Hierarchy  Process 
(AHP)  theory  to  generate  GIS  effectiveness  priorities.  Satty  [11]  introduced  AHP  and 
addressed  how  to  determine  the  relative  importance  of  a  set  of  activities  in  a  multi-criteria 
decision  problem.  AHP  has  been  applied  in  a  wide  variety  of  practical  settings  to  model 
complex  decision  problems.  AHP  makes  it  possible  to  incorporate  judgments  on 
intangible  qualitative  criteria  alongside  tangible  quantitative  criteria.  AHP  has  been  used 
in  ranking,  selection,  evaluation,  optimization,  and  prediction  decision  problems.  The 
process  utilizes  pairwise  comparisons  of  the  multiple  criteria  as  well  as  pairwise 
comparisons  of  the  multiple  sub-criteria.  The  use  of  such  comparisons  to  collect  data 
from  the  decision-maker  offers  significant  advantages.  It  allows  the  decision-maker  to 
focus  on  the  comparison  of  just  two  objects.  Additionally,  pairwise  comparisons  generate 
meaningful  information  about  the  decision  problem,  improve  consistency  in  the  decision 
making  process  [11]. 

The  AHP  structures  the  various  factors  or  criteria  of  the  decision  problem  into  an 
upside-down  tree  structure.  At  the  top  of  the  model  is  the  goal,  which  is  GIS 
effectiveness  in  crisis  management.  The  intermediate  level  in  the  tree  represents  the 
objectives  or  factors  to  take  into  consideration  when  making  a  specified  decision.  The 
GIS  objectives  or  factors  of  this  research  model  are  system  quality,  information  quality, 
user  satisfaction,  system  use,  decision  performance,  task  complexity  and  feedback,  and 
organizational  impact.  Another  level  of  sub-objectives  or  sub-factors  is  located  under  the 
objectives.  Figure  3  shows  the  built  model  by  the  Team  Expert  Choice,  which  also 
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shows  the  forty  GIS  sub-objectives.  The  definition  of  each  sub-factor  abbreviation  is 
shown  in  Table  1. 

Figure  3.  The  Team  Expert  Choice  Model  of  GIS  Effectiveness  in  Crisis  Management 
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Once  comparison  matrices  of  the  second  and  third  level  of  the  model  are  constructed, 
the  final  step  is  to  determine  the  overall  prioritization  of  the  objective  factors  of  GIS 
effectiveness  in  crisis  management.  Team  Expert  Choice  provides  a  9-prong  scale  as 
a  basis  for  comparison  between  any  two  elements.  They  are  as  follows: 

1  indicates  equal  importance  of  two  elements; 

3  indicates  moderate  importance  of  one  over  the  other; 


Table  1.  Definitions  of  the  Team  Expert  Choice  Model  Abbreviations 


Abbreviation 

Definition 

SYS-QAL 

SYSTEM  QUALITY 

1 

RESPONSE 

Response  /  turnaround  time 

2 

ERR-RECO 

Error  recovery 

3 

SEC-DATA 

Security  of  data  and  models 

4 

DOC-SYS 

Documentation  of  system  and  procedures 

5 

FLX-SYS 

Flexibility  of  the  system 

6 

IMP&EXP 

Import  and  export  data 

7 

DATA-MAN 

Data  manipulation 

8 

EASE-LEA 

Ease  of  learning 

9 

EASE-USE 

Ease  of  use 

INF-QAL 

INFORMATION  QUALITY 

10 

RELEVANCE 

Relevance 

11 

TIM-OUT 

Timeliness  of  output 

12 

PRS-OUT 

Precision  of  output 

13 

REL-OUT 

Reliability  of  output 

14 

CUR-OUT 

Currency  of  output 

15 

COM-OUT 

Completeness  of  output 

16 

CLR-OUT 

Clarity  of  output 

17 

PRES-MAP 

Presentation  mapping 

18 

VIEW-MAP 

Viewing  the  map 

USR-STF 

USER  SATISFACTION 

19 

TOP-  MAN 

Top  management  involvement 

20 

USR-PART 

User  participation 

21 

TRAINING 

Training  provided  to  user 

22 

USER-EXP 

User's  expectation  of  GIS  based  support 

23 

COMMUNIC 

Communication  between  users  &  technical  staff 

24 

RES-UTIL 

Resource  utilization 

25 

GIS  -POSI 

GIS  organizational  position 

SYS-USE 

SYSTEM  USAGE 

26 

TIME-COM 

Time  to  complete  a  task 

27 

FREQ-UAE 

Frequency  of  use 

28 

PERS-CON 

Personal  control  over  GIS  use 

29 

FUN-UTIL 

Number  of  GIS  functions  utilized 

DEC-MAC 

INDIVIDUAL  DECISION-MAKING 

30 

TIME-REQ 

Time  required  to  make  decisions 

31 

ACCU-DEC 

Accuracy  of  the  decision 

32 

DEC-CONF 

Decision  confidence 

TASK-COM 

TASK/FEEDBACK 

33 

#  OF  ALT 

Alternative  output  considered 

34 

UND-TASK 

Understanding  the  task  performed 

35 

PRE-TASK 

Predictability  of  task  results 

36 

FEEDBACK 

Feedback  from  manager/staff/ field 

ORG-IMP 

ORGANIZATIONAL  IMPACT 

37 

PRODUCTI 

Productivity  improved  by  GIS 

38 

PERFORMA 

Performance  improvement 

39 

COST-RED 

Cost  reduction 

40 

ACHEIVE 

Achieve  organization  goals 
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5  indicates  strong  importance  of  one  over  the  other; 

7  indicates  very  strong  importance  of  one  over  the  other; 

9  indicates  extreme  importance  of  one  over  the  other; 

and  2,  4,  6,  and  8  indicate  intermediate  values  between  two  adjacent  judgments. 

Since  there  are  no  alternatives  to  be  evaluated,  an  intensity  rating  scale  is  added.  This 

level  acts  as  an  intensity  scale  for  measuring  each  GIS  model  factor.  The  rating  scale 

for  this  exercise  is  Excellent,  Very  Good,  Good,  Fair,  and  Poor. 

3.1  The  Group  Decision  Making  Participants 

Six  experts  from  the  Information  Technology  Services  Directorate/Mapping  and 
Analysis  Center  at  the  Federal  Emergency  Management  Agency  (FEMA)  participated  in 
the  group  decision-making  exercise.  This  is  because  FEMA  is  the  federal  agency  that 
bears  primary  responsibility  for  the  nation’s  emergency  management  systems.  Moreover, 
all  participants  that  took  part  in  the  exercise  of  the  Team  Expert  Choice  model  of  GIS 
effectiveness  in  crisis  management  have  used  GIS  for  all  types  and  phases  of  crisis 
management.  The  positions  of  the  participants  in  the  Team  Expert  Choice  are: 

1.  Director  of  Mapping  and  Analysis  Center. 

2.  Team  leader  of  the  GIS  and  software  development  team. 

3.  Two  GIS  analysts. 

4.  GIS  technician. 

5.  Lead  geographer. 

The  researcher  acted  as  a  facilitator  when  using  Team  Expert  Choice  for  the 
group  of  the  decision-making  experts  of  FEMA.  The  researcher  was  responsible  for 
operating  the  program,  navigating  with  it,  and  directing  the  flow  of  discussion  for  the  at 
FEMA  during  the  exercise.  Each  expert  at  FEMA  treated  as  decision-makers.  They  had  a 
keypad  that  was  used  to  enter  their  judgments  in  the  system.  After  all  experts  have 
entered  all  judgments,  the  data  for  each  criterion  or  objective  is  added  together  and 
averaged  out.  The  average  is  then  entered  as  a  priority  for  that  objective.  After  all  of  the 
weights  have  been  recorded,  Team  Expert  Choice  generates  the  global  weights  of  the 
nodes  by  combining  the  local  priorities  throughout  the  entire  model. 

3.2  GIS  Model  Validation  and  Prioritization  Exercise  Results 

A  pretest  of  the  model  survey  was  conducted  with  FEMA  experts  to  make  sure 
that  those  measures  are  appropriate  in  terms  of  both  accuracy  and  feasibility.  After  that, 
some  minor  modifications  were  made  to  the  sub-objectives  and  its  measurements  in  the 
model.  Based  on  that  validation,  GIS  experts  conducted  39  pairwise  comparison 
assessments  through  the  help  of  the  facilitator  (researcher).  They  entered  their  judgments 
for  each  comparison  of  GIS  factors  (first  level)  and  sub-factors  (second  level).  Team 
Expert  Choice  calculated  the  geometric  mean  for  all  of  the  comparison  judgments  by  the 
panel  of  GIS  experts  for  each  objective  or  sub-objective.  For  each  set  of  the  pairwise 
comparison,  mathematical  calculations  were  performed  that  produce  priorities  (weights) 
and  included  a  measure  of  judgmental  consistency. 

Table  2  represents  the  assigned  local  priority  for  the  first  level  (seven  objectives) 
and  the  second  level  (forty  sub-objectives)  of  the  model  hierarchy.  For  the  first  hierarchy, 
factors  within  organizational  impact  were  the  most  important.  Organizational  impact 
received  a  weight  of  (0.504)  out  of  one.  This  means  that  organizational  impact  represents 
about  50%  of  the  importance  when  it  was  compared  with  the  other  six  factors  of  GIS 
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Table  2.  Local  Priorities  of  GIS  Effectiveness  Factors 


NO 

OBJECTIVE  (FACTOR) 

PRIORITY 

SYSTEM  QUALITY 

0.025 

1 

Response/turnaround  time 

0.048 

2 

Error  recovery 

0.026 

3 

Security  of  data  and  models 

0.055 

4 

Documentation  of  system  and  procedures 

0.105 

5 

Flexibility  of  the  system 

0.223 

6 

Import  and  export  data 

0.114 

7 

Data  manipulation 

0.107 

8 

Ease  of  learning 

0.128 

9 

Ease  of  use 

0.194 

INFORMATION  QUALITY 

0.054 

10 

Relevance 

0.069 

11 

Timeliness  of  output 

0.036 

12 

Precision  of  output 

0.054 

13 

Reliability  of  output 

0.159 

14 

Currency  of  output 

0.089 

15 

Completeness  of  output 

0.120 

16 

Clarity  of  output 

0.275 

17 

Presentation  mapping 

0.120 

18 

Viewing  the  map 

0.079 

USER  SATISFACTION 

0.077 

19 

Top  management  involvement 

0.028 

20 

User  participation 

0.057 

21 

Training  provided  to  user 

0.090 

22 

User's  expectation  of  GIS  based  support 

0.106 

23 

Communication  between  users  &  technical  staff 

0.217 

24 

Resource  utilization 

0.307 

25 

GIS  organizational  position 

0.193 

SYSTEM  USAGE 

0.042 

26 

Time  to  complete  a  task  (Short=Excel,  Long=Poor) 

0.305 

27 

Frequency  of  use  (Many=Excel,  Few=Poor) 

0.168 

28 

Personal  control  over  GIS  use 

0.272 

29 

Number  of  GIS  functions  utilized 

0.254 

INDIVIDUAL  DECESION-MAKING 

0.127 

30 

Time  to  make  decisions  (Short=Excel,  Long=Poor) 

0.189 

31 

Accuracy  of  the  decision 

0.490 

32 

Decision  confidence 

0.321 

TASK/FEEDBACK 

0.171 

33 

Alternative  output  considered  (Many=Excel,  Few=Poor) 

0.148 

34 

Understanding  the  task  performed 

0.324 

35 

Predictability  of  task  results 

0.240 

36 

Feedback  from  manager/staff/field 

0.288 

0.504 

37 

Productivity  improved  by  GIS 

0.190 

38 

Performance  improvement 

0.239 

39 

Cost  reduction 

0.151 

40 

Achieve  organization  goals 

0.420 

Total 

1.000 
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effectiveness  in  crisis  management.  The  second  most  important  group  of  factors  was  the 
task  complexity  and  feedback,  which  received  (0.171)  out  of  one.  This  indicates  that  task 
complexity  and  feedback  represents  about  17%  of  the  importance  when  compared  with 
the  other  factors  of  GIS  effectiveness  in  crisis  management.  The  third  most  important 
group  of  factors  was  individual  decision  making,  which  received  a  weight  of  (0.127)  out 
of  one.  This  means  individual  decision  making  represents  about  13%  of  the  importance 
when  compared  with  the  other  factors  of  GIS  effectiveness  in  crisis  management.  The 
fourth  most  important  group  of  factors  was  user  satisfaction,  which  received  a  weight  of 
(0.077)  out  of  one.  This  means  that  user  satisfaction  represents  about  8%  of  the 
importance  when  it  was  compared  with  the  other  factors  of  GIS  effectiveness  in  crisis 
management.  The  fifth  most  important  group  of  factors  was  information  quality,  which 
received  a  weight  of  (0.054)  out  of  one.  This  means  information  quality  represents  about 
5%  of  the  importance  when  compared  with  the  other  factors  of  GIS  effectiveness  in  crisis 
management.  The  sixth  most  important  group  of  factors  was  system  use,  which  received 
a  weight  of  (0.042)  out  of  one.  This  indicates  system  use  represents  about  4%  of  the 
importance  when  compared  with  the  other  factors  of  GIS  effectiveness  in  crisis 
management.  The  sixth  most  important  group  of  factors  was  system  use,  which  received 
a  weight  of  (0.042)  out  of  one.  This  indicates  system  use  represents  about  4%  of  the 
importance  when  compared  with  the  other  factors  of  GIS  effectiveness  in  crisis 
management.  And  the  last  group  of  factors  was  the  system  quality,  which  received  a 
weight  of  (0.025)  out  of  one.  System  quality  represents  about  3%  of  the  importance  of  the 
GIS  effectiveness  factors. 

The  global  priority  of  the  second  level  of  the  hierarchy  was  calculated  and  ranked 
by  sorting  from  the  lowest  to  the  highest  as  shown  in  Table  3.  The  pairwise  comparison 
judgments  indicated  that  all  participants  agreed  on  the  Response/turnaround  time,  error 
recovery,  and  security  of  data  and  models  have  the  lowest  priority  in  the  model  of  0.001. 
They  ranked  the  least  important  when  compared  with  the  other  thirty-seven  GIS  factors. 
However,  the  highest  priorities  of  GIS  factors  in  the  model  achieved  were  organization 
goals,  perfonnance  improvement,  and  productivity  improved  by  GIS  and  have  weights  of 
0.21 1,  0.121,  and  0.096  respectively  and  ranked  the  most  important. 

3.3  Establishing  Intensity  Scale  for  Research  Instrument 

Since  there  are  no  alternatives  to  be  evaluated,  an  intensity  scale  was  created  to 
rate  or  assess  the  forty  GIS  effectiveness  sub-objectives  or  sub-factors  in  the  GIS 
instrument  survey.  The  GIS  instrument  is  based  on  the  theoretical  and  validated  GIS 
effectiveness  model.  In  constructing  intensity  scales,  the  intention  should  be  made  to 
convey  the  information  about  the  sub-objective  being  rated  and  to  use  ranges  that  are 
logical  to  the  evaluator.  The  ratio  ranges  of  the  intensity  scale  do  not  have  to  be  equal  in 
span.  The  ratio  scale  values  should  represent  the  reality  found  in  the  work  environment. 
The  GIS  panel  of  experts  at  FEMA  participated  to  derive  a  new  customized  intensity  ratio 
scale  by  prioritizing  the  intensity  nodes  through  the  usual  pairwise  comparison  process  in 
Team  Expert  Choice.  After  being  prioritized,  they  may  be  thought  of  as  a  measurement 
scale  for  each  GIS  sub-objective. 
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Table  3.  Global  Priority  of  GIS  Effectiveness  Factors 


NO 

SUB-OBJECTIVE 

PRIORITY 

l 

Response/turnaround  time 

0.001 

2 

Error  Recovery 

0.001 

3 

Security  of  data  and  models 

0.001 

4 

Timeliness  of  output 

0.002 

5 

Top  management  involvement 

0.002 

6 

Documentation  of  system  procedures 

0.003 

7 

Import  and  export  data 

0.003 

8 

Data  manipulation 

0.003 

9 

Ease  of  learning 

0.003 

15 

Currency  of  output 

0.005 

16 

Flexibility  of  the  system 

0.006 

17 

Completeness  of  output 

0.006 

18 

Presentation  mapping 

0.006 

19 

Training  provided  to  user 

0.007 

20 

Frequency  of  use 

0.007 

21 

User's  expectation  of  GIS  based  support 

0.008 

22 

Reliability  of  output 

0.009 

23 

Number  of  GIS  functions  utilized 

0.011 

24 

Personal  control  over  GIS  use 

0.012 

25 

Time  to  complete  a  task 

0.013 

26 

Clarity  of  output 

0.015 

27 

GIS  organizational  position 

0.015 

28 

Communication  between  users  &  technical  staff 

0.017 

29 

Resource  utilization 

0.024 

30 

Time  required  to  make  decisions 

0.024 

31 

Alternative  output  considered 

0.025 

32 

Decision  confidence 

0.041 

33 

Predictability  of  task  results 

0.041 

34 

Feedback  from  manager/staff/field 

0.049 

35 

Understanding  the  task  performed 

0.055 

36 

Accuracy  of  the  decision 

0.062 

37 

Cost  reduction 

0.076 

38 

Productivity  improved  by  GIS 

0.096 

39 

Performance  improvement 

0.121 

40 

Achieve  organization  goals 

0.211 

The  FEMA  experts  identified  and  prioritized  a  scale  with  five  distinct  ratings. 
These  ratings  were  Excellent  (0.561),  Very  good  (0.253),  Good  (0.1 18),  Fair  (0.045),  and 
poor  (0.023).  Once  the  model  is  used  for  rating  instead  of  alternatives,  the  global  weights 
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of  the  intensity  scale  for  rating  the  GIS  factors  were  transferred  to  a  normalized  scale 
from  one  to  zero.  The  new  weights  of  the  normalized  intensity  scale  are  listed  in  the 
Table  4. 


Table  4.  Intensity  Scale  Weights 


Intensity 

Intensity 

Normalized 

Scale 

Priority 

Priority 

Excellent 

0.561 

1.00 

Very  Good 

0.253 

0.45 

Good 

0.118 

0.21 

Fair 

0.045 

0.08 

Poor 

0.023 

0.04 

4.  GIS  Effectiveness  Factors  Assessments 
4.1  Survey  Instrument 

Based  on  the  theoretical  and  validated  GIS  model,  the  survey  instrument  was 
developed.  The  survey  consists  of  two  parts.  The  first  part  is  to  collect  demographic  data 
that  includes  organization  type,  organization  size,  GIS  software,  crisis  phases,  the 
respondent  is  position,  GIS  work  experience,  and  GIS  training  duration.  The  second  part 
of  the  survey  is  to  assess  the  forty  sub-factor  of  GIS  effectiveness  in  crisis  management. 
The  assessments  were  based  on  the  intensity  scale  identified  by  FEMA  experts  as 
explained  above,  and  ranged  from  excellent  being  (1.0)  to  poor  being  (0.04).  Moreover, 
the  survey  instrument  was  pre-tested  by  FEMA  experts. 

A  cross  sectional  survey  was  used  to  collect  data  at  one  point  in  time  from  a 
sample  selected  to  describe  some  larger  population  at  that  time.  In  order  to  choose  a 
research  sample  that  properly  reflects  the  entire  population  under  consideration,  a  random 
sampling  procedure  was  utilized.  Fifty-four  government  and  private  organizations 
participated  in  this  research  selected  from  FEMA  and  the  United  States  Geological 
Survey  (USGS)  email  list.  The  sample  subjects  consisted  of  GIS  directors,  GIS  project 
managers,  GIS  technical  staff,  and  GIS  users.  The  technical  staff  includes  system 
administrators,  team  leaders,  programmers,  and  system  analysts.  The  population  sample 
needed  to  have  hands-on  experience  with  GIS  applications,  for  small-scale  emergencies 
such  as  fire  or  chemical  incidents  or  large-scale  disasters  such  as  tornado,  flood  or 
hurricane  events. 

Three  methodologies  were  used  to  distribute  the  survey.  The  first  was  through  the 
distribution  of  the  survey  in  GIS  conferences  and  seminars.  The  second  was  through  the 
mail.  And  the  third  was  through  a  web  page  developed  for  the  survey  using  the  Cold 
Fusion  Server  software  version  4.0.  The  data  collected  through  the  web  was  linked  to 
ACCESS  database,  and  the  researcher  was  able  to  view  the  data.  The  URL  address  for 
the  survey  web  page  was  sent  to  the  subjects  by  email.  The  email  addresses  of  the 
subjects  were  obtained  from  the  United  States  Geological  Survey  (USGS)  email  list  of 
GIS  professionals  in  crisis  and  disaster  application.  Other  email  addresses  were  obtained 
from  GIS  conferences  and  seminars.  One  thousand  one  hundred  and  thirty  four  surveys 
were  distributed  by  the  means  of  the  above  three  methods.  One  hundred  eighteen 
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completed  survey  forms  were  collected.  Eighty-seven  survey  forms  were  collected  by  the 
use  of  the  web  page  and  the  thirty-one  were  collected  by  the  other  two  methods.  Ninety- 
eight  answered  survey  data  forms  were  used  for  the  analysis  and  twenty  collected  survey 
forms  were  discarded  due  to  missing  data  or  being  unusable.  The  data  collected  from  the 
survey  were  analyzed  to  measure  and  assess  the  GIS  factors  effectiveness  in  crisis 
management  (third  stage),  and  to  investigate  any  significant  differences  among  survey 
respondents  as  independent  variables  in  term  of  their  position,  level  of  experience,  and 
training  in  the  effective  utilization  of  GIS  factors  as  dependant  variables,  through  the 
formulated  research  hypotheses  (fourth  stage). 

4.2  Reliability  Analysis 

Reliability  can  be  estimated  from  empirical  data.  The  reliability  of  the  instrument 
is  concerned  with  its  consistency.  This  research  used  the  Cronbach’s  alpha  value  in  order 
to  assess  the  internal  consistency  of  the  results  across  items  within  a  test.  Alpha  values 
above  70%  are  acceptable  indicators  of  internal  consistency  [12].  Alpha  values  were 
calculated  for  each  sub-scale.  Table  5  shows  the  value  of  the  reliability  test  for  the  main 
seven  factors  of  the  GIS  effectiveness  in  crisis  management. 

From  the  table,  one  may  see  that  the  two  lower  reliability  scores  are  system  use 
and  task  complexity  and  feedback.  The  corresponding  reliabilities  were  74%  and  79%. 
They  are  the  weakest  in  the  results  nevertheless  they  pass  the  benchmark  of  70%  that  is 
customarily  used  for  internal  consistency.  Moreover,  the  questions  that  were  designed 
under  those  two  factors  were  developed  by  the  researcher  and  have  not  been  widely  used 
to  measure  effectiveness  of  other  types  of  information  systems  when  compared  with  the 
other  five  factors.  In  addition,  of  the  98  respondents,  only  84  of  them  had  complete 
answers  to  the  40  items.  There  were  14  subjects  chose  not  to  answer  some  questions  that 
explains  the  variability  of  the  sample  size  within  each  sub-scale. 

4.3  Demographic  Variables  Analysis 

Table  6  shows  the  analysis  of  the  demographic  variables.  These  variables  are: 
organization  types  (government  or  private),  organization  size  (large  to  medium  or  small), 
GIS  software  (Arclnfo,  ArcView,  MGE,  Maplnfo,  ERDAS,  and  others),  crisis  type 
(natural,  technological,  human  made,  and  others),  crisis  phase  (preparation,  response, 
recovery,  and  mitigation),  the  respondent  position  (GIS  directors  or  project  manager,  GIS 
technical  staff,  and  GIS  user),  GIS  work  experience  (Less  than  2  years,  between  2  to  6 
years,  and  more  than  6  years),  and  GIS  training  duration  (Less  than  6  months,  between  6 
to  12  months,  and  more  than  a  year). 

4.4  Survey  Data  Analysis 

Table  7  shows  the  results  of  the  average  assessments  and  the  calculated  standard 
deviation  for  GIS  factors  from  the  ninety-eight  survey  participants  with  their  relative 
weights  that  were  obtained  from  the  GIS  experts.  The  assessment  of  actual  systems 
results  for  all  of  the  survey  participants  indicates  that  low  assessments  (averages  less  than 
0.30)  were  given  to  the  following  sub-factors;  error  recovery  (0.29),  documentation  of 
system  and  procedures  (0.26),  ease  of  learning  (0.27),  currency  of  output  (0.28),  top 
management  involvement  (0.29),  training  provided  to  user  (0.28),  and  GIS  organizational 
position  (0.25).  Further  investigation  needs  to  be  conducted  in  these  areas  for  GIS 
applications  in  crisis  management.  Higher  assessments  (averages  more  than  0.45)  were 
given  to  the  sub-factors  of  presentation  mapping  (0.49),  viewing  the  map  (0.49),  and 
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productivity  improved  by  GIS  (0.47).  This  indicates  that  all  survey  participants  agreed 
that  those  areas  were  very  well  developed  in  GIS.  Moreover,  user  satisfaction  factor  were 
assessed  the  lowest  (0.30)  and  organizational  impact  were  assessed  the  highest  (0.44). 


Table  5.  Reliability  Analysis  of  the  GIS  Effectiveness  Factors 


GIS 

Sub-Scale 

Priority 

(%) 

No.  of 
Items 

Sample 

Size 

Reliability 

System 

Quality 

2.5 

9 

84 

0.88 

Information 

Quality 

9 

88 

0.91 

User 

Satisfaction 

7.7 

7 

91 

0.84 

System 

Usage 

4.2 

4 

94 

0.74 

Individual 
Decision  Making 

12.7 

3 

88 

0.85 

Task  Complexity 
/  Feedback 

17.1 

4 

88 

0.79 

Organizational 

Impact 

50.4 

4 

86 

0.93 

All 

Objectives 

1.00 

40 

84 

0.97 
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Table  6.  Demographic  Variables  Analysis 


NO. 

ITEM 

SAMPLE 

SIZE 

Percentage 

1.  A 

GOVERNMENT 

72 

73% 

ORGNIZ  ATION  S 

B 

PRIVATE  ORGANIZATIONS 

26 

27% 

2.  A 

MEDUIM  TO  LARGE 

76 

78% 

BUSINESS 

B 

SMALL  BUSINESS 

22 

22% 

3.  A 

ARCINFO 

53 

54% 

B 

ARCVIEW 

72 

73% 

C 

MGE 

15 

15% 

D 

MAPINFO 

33 

34% 

E 

ERDAS 

17 

17% 

F 

OTHER  GIS  SOFTWARE 

37 

38% 

4.  A 

NATURAL  DISASTER 

87 

89% 

B 

TECHNOLOGICAL  CRISIS 

51 

52% 

C 

HUMAN  CRISIS 

44 

45% 

D 

OTHER  TYPES  OF  CRISIS 

23 

23% 

5.  A 

CRISIS  PREPAREDNESS 

73 

74% 

B 

CRISIS  RESPONSE 

79 

81% 

C 

CRISIS  RECOVERY 

60 

61% 

D 

CRISIS  MITIGATION 

51 

52% 

6.  A 

DIRECTORS  &  PROJECT 

41 

42% 

MANAGERS 

B 

TECHNICAL  PERSONAL 

37 

38% 

C 

USERS 

20 

20% 

7  A 

GIS  WORK  EXPERIENCE 

1  9 

12% 

LESS  THAN  2  YEARS 

X  Z 

GIS  WORK  EXPERIENCE 

97 

37% 

JtS 

BETWEEN  2  TO  6  YEARS 

9  / 

GIS  WORK  EXPERIENCE 

/IQ 

50% 

MORE  THAN  6  YEARS 

8.  A 

GIS  TRAINING 

30 

31% 

LESS  THAN  6  MONTHS 

R 

GIS  TRAINING 

1  7 

1  7% 

D 

BETWEEN  6-12  MONTHS 

1  / 

1  /  /o 

P 

GIS  TRAINING 

C  1 

MORE  THAN  1  YEAR 

9  1 

52% 

Table  7.  GIS  Effectiveness  Factors  Assessments 
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NO. 

OBJECTIVE  (FACTOR) 

PRIORITY 

ASSESSMENT 

MEAN 

SD 

SYSTEM  QUALITY 

0.025 

0.35 

1 

Response/ turnaround  time 

0.001 

0.42 

0.21 

2 

Error  recovery 

0.001 

0.29 

0.19 

3 

Security  of  data  and  models 

0.001 

0.32 

0.21 

4 

Documentation  of  system  and  procedures 

0.003 

0.26 

0.17 

5 

Flexibility  of  the  system 

0.006 

0.33 

0.19 

6 

Import  and  export  data 

0.003 

0.40 

0.24 

7 

Data  manipulation 

0.003 

0.42 

0.23 

8 

Ease  of  learning 

0.003 

0.27 

0.17 

9 

Ease  of  use 

0.005 

0.31 

0.22 

INFORMATION  QUALITY 

0.054 

0.38 

10 

Relevance 

0.004 

0.40 

0.20 

11 

Timeliness  of  output 

0.002 

0.37 

0.17 

12 

Precision  of  output 

0.003 

0.32 

0.19 

13 

Reliability  of  output 

0.009 

0.33 

0.19 

14 

Currency  of  output 

0.005 

0.28 

0.18 

15 

Completeness  of  output 

0.006 

0.30 

0.20 

16 

Clarity  of  output 

0.015 

0.39 

0.24 

17 

Presentation  mapping 

0.006 

0.49 

0.23 

18 

Viewing  the  map 

0.004 

0.49 

0.24 

USER  SATISFACTION 

0.077 

0.30 

19 

Top  management  involvement 

0.002 

0.29 

0.18 

20 

User  participation 

0.004 

0.39 

0.19 

21 

Training  provided  to  user 

0.007 

0.28 

0.19 

22 

User's  expectation  of  GIS  based  support 

0.008 

0.34 

0.21 

23 

Communication  between  users  &  technical  staff 

0.017 

0.30 

0.18 

24 

Resource  utilization 

0.024 

0.30 

0.19 

25 

GIS  organizational  position 

0.015 

0.25 

0.17 

SYSTEM  USAGE 

0.042 

0.36 

26 

Time  to  complete  a  task  (Short=Excel,  Long=Poor) 

0.013 

0.31 

0.18 

27 

Frequency  of  use  (Many=Excel,  Few=Poor) 

0.007 

0.39 

0.21 

28 

Personal  control  over  GIS  use 

0.012 

0.39 

0.23 

29 

Number  of  GIS  functions  utilized 

0.011 

0.38 

0.21 

DECISION  PERFORMANCE 

0.127 

0.37 

30 

Time  to  make  decisions  (Short=Excel,  Long=Poor) 

0.024 

0.33 

0.22 

31 

Accuracy  of  the  decision 

0.062 

0.38 

0.21 

32 

Decision  confidence 

0.041 

0.37 

0.22 

TASK / FEEDBACK 

0.171 

0.35 

33 

Alternative  output  considered  (Many=Excel,  Few=Poor) 

0.025 

0.33 

0.21 

34 

Understanding  the  task  performed 

0.055 

0.34 

0.18 

35 

Predictability  of  task  results 

0.041 

0.35 

0.20 

36 

Feedback  from  manager/ staff/ field 

0.049 

0.37 

0.18 

ORGANIZATIONAL  IMPACT 

0.504 

0.44 

37 

Productivity  improved  by  GIS 

0.096 

0.47 

0.26 

38 

Performance  improvement 

0.121 

0.44 

0.23 

39 

Cost  reduction 

0.076 

0.38 

0.21 

40 

Achieve  organization  goals 

0.211 

0.42 

0.24 

Total 

1.000 
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5.  Inferential  Analysis  of  the  Data 

Inferential  analysis  was  the  fourth  stage  in  this  research.  Inferential  analysis  of  the 
data  was  conducted  to  examine  the  five  null  research  hypotheses  derived  from  the 
research  hypotheses  and  was  based  on  the  parametric  statistic  one-way  analysis  of 
variance  (ANOVA).  Prior  to  applying  (ANOVA)  for  each  hypothesis,  the  validity  of  the 
assumptions  associated  with  this  statistic  was  examined  and  satisfied  for  the  seven  GIS 
effectiveness  factors.  These  assumptions  are  normality  of  the  distribution,  independence 
of  the  observations,  and  Homogeneity  of  the  variance.  As  far  the  homogeneity  of  the 
variance,  the  Levene  test  was  utilized.  Once  ANOVA  determines  that  the  effect  of  some 
independent  variable  is  significant  which  indicates  that  differences  exist  among  the 
means,  Tukey  test  of  post  hoc  testing  is  used  to  make  pairwise  multiple  comparisons  that 
can  determine  which  means  differ.  The  null  hypothesis  is  rejected  when  the  likelihood  of 
the  observed  difference  in  means  occurring  by  chance  (reported  by  the  p-value)  is  less 
than  0.05,  which  is  the  level  of  significance.  In  those  cases  we  assume  there  is  a 
statistically  significance  difference  between  the  two  means. 

Research  Hypothesis  1:  There  are  significant  differences  in  GIS  effectiveness  in  crisis 
management  among  GIS  directors  and  project  managers,  GIS  technical  staff,  and  GIS 
users  as  perceived  in  system  quality,  information  quality,  user  satisfaction,  system  use, 
individual  decision  making,  task  complexity  and  feedback,  and  organizational  impact. 
Test  Findings:  According  to  the  test  result  presented,  there  is  no  significant  difference 
among  GIS  position  subgroups  in  the  areas  of  system  quality,  information  quality,  user 
satisfaction,  and  individual  decision  making.  The  null  hypothesis  was  retained;  the 
analyzed  data  did  not  provide  sufficient  evidence  to  reject  the  null  hypothesis  for  those 
specific  four  factors  of  GIS  effectiveness  in  crisis  management.  However,  the  null 
hypotheses  were  rejected  in  the  other  three  areas  of  GIS  effectiveness  in  crisis 
management:  system  use,  task  complexity/feedback,  and  organizational  impact.  The 
analyzed  data  provides  sufficient  evidence  for  a  significant  difference  between  the  GIS 
technical  staff  and  GIS  users.  The  technical  staffs  have  significantly  higher  assessments 
than  users  as  perceived  in  system  use,  task  complexity/feedback,  and  organizational 
impact  in  GIS  effectiveness  in  crisis  management  as  explained  below. 

Test  Analysis: 

A)  System  use:  Levene’ s  test  indicates  that  the  homogeneity  of  the  variance  for  all  of  the 
three  tested  GIS  positions  is  met.  Therefore,  the  ANOVA  was  utilized  for  the  analysis. 
The  ANOVA  results  are  F=  3.963  and  p=  0.022<0.05,  which  indicates  that  there  is  a 
statistical  significant  differences  among  the  three  subgroup  of  GIS  positions  in  the 
effective  utilization  of  GIS  in  crisis  management.  Further  investigation  was  needed  to 
find  which  subgroup  is  different.  This  was  achieved  by  utilizing  the  post  hoc  Tukey  test. 
The  test  reported  a  significant  difference  level  of  0.020<0.05  between  GIS  users  with  a 
reported  mean  0.26  and  GIS  technical  staff  with  a  reported  mean  0.41,  but  the  GIS 
directors  and  project  managers  subgroup  with  a  reported  mean  0.38  does  not  differ  from 
either  with  a  significant  level  of  0.788>0.05  with  technical  staff  and  0.069>0.055  with 
users.  This  could  be  interpreted  as  the  technical  personnel  are  more  at  ease  and  aware  in 
utilizing  the  GIS  capabilities  for  crisis  applications  than  the  end  users. 

B)  Complexity/ feedback:  The  homogeneity  of  the  variance  from  Levene’ s  test  for  all  of 
the  three  tested  GIS  positions  subgroups  shows  p=  0.007<0.05,  which  indicates  a 
heterogeneity  of  the  variance.  Therefore,  the  analog  non-parametric  Kruskal- Wallis  test 
was  used  instead  of  the  parametric  ANOVA  test.  The  chi-square  test  results  show 
significant  differences  among  the  three  subgroups  of  GIS  positions  in  the  task  complexity 
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and  feedback  since  p=  0.021<0.05.  Therefore,  further  investigation  was  needed  to  find 
which  subgroup  is  different.  This  is  achieved  by  utilizing  the  post  hoc  Tukey  test.  The 
test  reported  a  significant  difference  in  GIS  task  complexity/feedback  with  a  level  of 
0.015  <  0.05  between  GIS  users  with  a  reported  mean  0.25  and  GIS  technical  staff  with  a 
reported  mean  0.43.  However,  the  GIS  directors  and  project  managers  subgroup  with  a 
reported  mean  0.34  does  not  differ  from  technical  staff  and  users  with  a  significant  level 
of  0.271>0.05  with  technical  staff  and  0.278>0.055  with  users.  This  might  be  attributed 
to  the  fact  that  users  have  less  experience  with  GIS  tasks  in  crisis  applications  and  less 
understanding  of  the  power  of  the  system  than  the  technical  personnel. 

C)  Organizational  impact:  The  homogeneity  of  the  variance  from  Levene’s  test  for  all  of 
the  three  tested  GIS  positions  subgroups  shows  p=0.036<0.05,  which  indicates  a 
heterogeneity  of  the  variance.  Therefore,  the  analog  non-parametric  Kruskal- Wallis  test 
was  used  instead  of  the  parametric  ANOVA  test.  The  chi-square  test  results  show 
significant  differences  among  the  three  subgroups  of  GIS  positions  in  the  organizational 
impact  since  p=  0.047<0.05.  Therefore,  further  investigation  was  needed  to  find  which 
subgroup  is  different.  This  was  achieved  by  utilizing  the  post  hoc  Tukey  test.  The  test 
reported  a  significant  difference  in  GIS  organizational  impact  with  a  level  of  0.035<0.05 
between  GIS  users  with  a  reported  mean  0.31  and  GIS  technical  staff  with  a  reported 
mean  0.51.  However,  GIS  directors  and  project  managers  subgroup  with  a  reported  mean 
0.45  does  not  differ  from  users  or  technical  staff  with  a  significant  level  of  0.694>0.05 
with  technical  staff  and  0. 14 1>0.05  with  users.  This  could  be  argued  that  users  represent 
the  lower  managerial  level  in  their  organizations,  which  makes  them  unaware  of  the 
indirect  positive  impact  of  GIS  on  the  organizations  when  compared  with  the  technical 
personnel. 

Research  Hypothesis  2:  There  are  significant  differences  in  GIS  effectiveness  in  crisis 
management  among  GIS  users  based  on  their  years  of  work  experience  as  perceived  in 
system  quality,  infonnation  quality,  user  satisfaction,  system  use,  individual  decision 
making,  task  complexity  and  feedback,  and  organizational  impact. 

Test  Findings:  According  to  the  test  result  presented,  there  is  only  significant  difference 
among  GIS  experience  subgroups:  beginners  (less  than  2  years),  professionals  (between 
2-6  years),  and  experts  (more  than  six  years)  in  the  effectiveness  of  GIS  system  quality  in 
crisis  management  and  the  null  hypothesis  was  rejected.  The  analyzed  data  provides 
sufficient  evidence  for  a  significant  difference  between  the  GIS  experts  and  GIS 
professionals.  The  experts  are  significantly  higher  assessed  the  GIS  than  professionals  in 
their  perception  of  the  system  quality  effectiveness  in  crisis  management.  However,  the 
null  hypothesis  was  retained;  the  analyzed  data  did  not  provide  sufficient  evidence  to 
reject  the  null  hypothesis  for  the  other  six  factors  of  GIS  as  explained  below. 

Test  Analysis: 

A)  System  quality:  The  homogeneity  of  the  variance  from  Levene’s  test  for  all  of  the 
three  tested  GIS  users  experience  subgroups  shows  p=0.001<0.05,  which  indicates  a 
heterogeneity  of  the  variance.  Therefore,  the  analog  non-parametric  Kruskal- Wallis  test 
was  used  instead  of  the  parametric  ANOVA  test.  The  chi-square  test  results  show 
significant  differences  among  the  three  subgroups  of  GIS  positions  in  the  system  quality 
since  p=  0.038<0.05.  Therefore,  further  investigation  was  needed  to  find  which  subgroup 
is  different.  This  is  achieved  by  utilizing  the  post  hoc  Tukey  test.  The  test  did  not  show  a 
significant  difference  among  the  three  subgroup  of  GIS  experience  in  system  quality. 
Therefore,  the  K-W  test  results  utilized  which  report  a  significant  difference  between  the 
subjects  who  are  experts  (more  than  6  years  of  experience)  with  a  reported  mean  rank  of 
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48  and  those  who  are  professionals  (between  2-6  years  of  experience)  with  a  reported 
mean  rank  of  34.  However,  those  who  are  beginners  (less  than  2  years  of  experience) 
subgroup  with  a  reported  mean  rank  of  43  did  not  differ  from  experts  or  professionals. 
This  might  be  interpreted  that  the  GIS  experts  have  more  experience  with  GIS  software 
features  and  capabilities  and  feel  more  confident  in  the  software  quality  for  crisis 
applications  than  the  GIS  professionals. 

Research  Hypothesis  3:  There  are  significant  differences  in  GIS  effectiveness  in  crisis 
management  among  GIS  users  based  on  duration  of  training  as  perceived  in  system 
quality,  information  quality,  user  satisfaction,  system  use,  individual  decision  making, 
task  complexity  and  feedback,  and  organizational  impact. 

Test  Findings:  According  to  the  test  result  presented,  there  is  no  significant  difference 
among  GIS  users  training  subgroups  (less  than  6  months,  between  6  to  12  months,  and 
more  than  12  months)  in  the  effectiveness  of  GIS  in  crisis  management  as  perceived  in 
system  quality,  infonnation  quality,  user  satisfaction,  system  use,  individual  decision¬ 
making,  task  complexity/  feedback,  and  organizational  impact.  The  analyzed  data  did  not 
provide  sufficient  evidence  to  reject  the  null  hypothesis.  Table  8  provides  a  summary  of 
the  five  research  hypotheses  testing  findings. 


Table  8.  Summary  of  the  Research  Hypotheses  Testing  Findings 


Ind. Variable 

Ded.Variable 

(HI) 

Position 

[Directors& 
project  managers, 
technical,  or 
users] 

(H2) 

Work 

Experience 

[<2,  between 

2  to  6,  or  >6 
Years] 

(H3) 

Duration  of 
Training 

[<  6,  between  6 
tol2,  or  >12 
Months] 

System  Quality 

Not  Significant 

Significant 

Not  Significant 

Infonnation  quality 

Not  Significant 

Not 

Significant 

Not  Significant 

User  satisfaction 

Not  Significant 

Not 

Significant 

Not  Significant 

System  use 

Significant 

Not 

Significant 

Not  Significant 

Decision  making 

Not  Significant 

Not 

Significant 

Not  Significant 

Task  complexity/ 
Feedback 

Significant 

Not 

Significant 

Not  Significant 

Organization-al 

Impact 

Significant 

Not 

Significant 

Not  Significant 
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6.  Operation  and  Policy  Recommendations 

Based  on  this  research,  it  is  possible  to  make  a  number  of  recommendations  that 
should  result  in  more  effective  utilization  of  GIS.  The  following  recommendations  come 
directly  from  the  research: 

1.  GIS  managers  should  regularly  measure  the  GIS  effectiveness  for  each  project  or 
crisis  events  and  establish  a  log.  At  the  same  time,  they  may  consider  other 
technical,  individual  and  organizational  factors  to  be  measured  in  the  model 
according  to  the  crisis  context.  Therefore,  the  log  data  will  show  how  the  crisis 
events  or  projects  are  being  managed.  Additionally,  the  above  measurements  will 
improve  the  effectiveness  of  GIS,  which  will  affect  on  the  organizational 
performance. 

2.  The  decision-making  model  of  GIS  effectiveness  was  used  for  the  crisis 
applications  domain.  This  is  to  avoid  any  potential  for  distorted  or  biased  data  by 
using  GIS  for  different  domains  of  applications.  The  model  can  be  used  for  other 
GIS  applications  such  as  transportation,  urban  planning,  telecommunication  and 
so  forth. 

3.  Future  development  and  refinement  of  measurements  are  needed.  It  is  important 
to  highlight  the  need  to  review  the  appropriateness  of  the  GIS  measurements 
according  to  GIS  future  capabilities,  crisis  context,  and  organization  needs. 

7.  Recommendations  for  Further  Study 

1.  This  study  captured  infonnation  from  the  GIS  population  in  crisis  management 
that  includes  all  levels  or  classes  of  employees.  This  study  surveyed  GIS  directors 
and  GIS  project  managers,  GIS  technical  staff,  and  GIS  users.  While  managers 
surely  represent  a  major  group  within  organizations  using  GIS  in  this  study,  users 
are  equally  important.  GIS  end-users  represented  the  smallest  sample  size  in  this 
study.  Therefore,  future  research  that  targets  this  class  of  employees  and 
compares  them  to  managers  is  clearly  needed.  Such  research  is  important  in 
furthering  our  understanding  of  the  myriad  issues  that  have  been  addressed  herein 
in  terms  of  user  satisfaction. 

2.  For  the  four  crisis  management  phases  of  preparedness,  response,  recovery, 
mitigation,  some  respondents  replied  to  all  different  four-crisis  phases.  Therefore, 
the  independence  of  the  subjects  was  lacking.  Therefore,  the  repeated  multi 
variant  analysis  of  variance  (MANOVA)  procedure  could  be  used,  provided  there 
was  sufficient  sample  size.  However,  there  were  not  enough  observations  or 
respondents  for  the  four-crisis  phases  simultaneously  to  use  MANOVA  to  find 
any  significant  differences  in  GIS  effectiveness  in  crisis  management  among  the 
four  phases  of  crisis  management. 

3.  User  satisfactions  factors  were  assessed  the  lowest  among  the  other  six  factors  of 
GIS.  More  specific  research  is  recommended  to  investigate  the  GIS  users 
satisfaction  in  the  field  of  crisis  management.  Additionally,  a  comparative  study 
may  needed  to  find  how  GIS  users  differ  from  other  information  systems 
applications  users  that  may  add  value  to  the  GIS  research  literature. 

4.  Another  way  of  doing  this  study  would  be  to  allow  the  GIS  users  to  prioritize  the 
GIS  effectiveness  model  and  have  experts  assess  the  model  factors  through  a 
survey.  Then  a  comparative  study  will  be  needed  to  compare  the  model  evaluation 
from  the  users’  point  of  view  and  the  experts’  point  of  view. 
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5.  In  order  to  achieve  effective  GIS  technology  and  to  capture  the  potential  benefits 
from  employing  the  new  technology,  the  factors  associated  with  it  should  be 
studied  at  various  future  stages  of  growth  and  development  of  GIS.  Other  system 
technical  features  and  the  consideration  of  particular  organizational  functions  in 
relation  to  GIS  implementation  could  also  be  incorporated  in  future  research. 

6.  Team  Expert  Choice  that  employs  the  Analytical  Hierarchy  Process  does  not 
perform  pairwise  comparison  between  sub-factors  that  are  not  related  to  the  same 
factor  by  assuming  that  those  sub-factors  are  independent  from  each  other. 
However,  there  are  some  relation  between  the  GIS  sub-factors  that  are  not  relate 
to  the  same  factor  such  easy  to  use  sub-factor  and  training  provided  to  user  sub¬ 
factor.  Therefore,  another  decision  making  technique  may  be  considered  for 
developing  GIS  effectiveness  model. 

7.  This  study  did  not  address  specific  GIS  software  features  as  a  sub-factor  of  the 
system  quality  factor.  Team  Expert  Choice  (educational  version)  allows  no  more 
than  nine  nodes  (factors)  in  each  level,  which  then  forces  the  nodes  to  the  next 
lower  level.  Thus,  future  studies  may  include  these  GIS  features  such  as  polygon 
overlay,  networking,  and  so  forth  under  the  sub-factor  of  GIS  data  manipulation. 

8.  Concluding  Remarks 

GIS  is  a  spatial  information  system  that  needs  to  be  managed  properly  in  order  to 
be  used  effectively.  The  GIS  development  and  evaluation  research  that  has  been 
published  to  date  in  the  literature  by  information  systems  researchers  has  concentrated 
primarily  on  public  sector  organizations  and  on  measuring  technical  system  features 
rather  than  individual  behavior  and  organizational  factors  that  affecting  success  [20]. 
Moreover,  there  is  not  a  systematic  study  of  GIS  evaluation  across  multiple  organizations 
that  has  been  published  [20]. 

This  study  presents  the  first  research  that  focused  on  technical,  as  well  as 
individual,  and  organizational  factors  that  affects  GIS  success.  More  importantly,  this 
study  may  be  considered  the  first  empirical  study  that  introduces  a  systematic  model  to 
measure  GIS  effectives,  and  that  surveyed  across  multiple  public  and  private 
organizations. 

GIS  is  rapidly  becoming  an  integral  part  of  the  crisis  management.  GIS  managers 
work  under  a  great  deal  of  pressure  in  case  of  a  crisis.  Their  line  of  work  demands  high 
levels  of  precision  delivered  in  the  least  amount  of  time.  If  GIS  variables  are  managed 
effectively,  the  potential  exists  to  reduce  cost,  shorten  time  of  planning,  improve  crisis 
preparedness,  response,  recovery,  and  mitigation  operations,  and  reduce  organization 
performance  deficiencies. 

Moreover,  the  crisis  management  field  may  be  now  mature  enough  to  use 
knowledge  gained  from  GIS  effectiveness  measurements  and  leverage  this  knowledge  in 
improving  the  quality  of  existing  GIS  technology  and  periodically  examine  its  use  in  the 
field. 
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